(opt-in) use curl for tracker communication by TheBaronVladimirHarkonnen · Pull Request #8159 · arvidn/libtorrent

TheBaronVladimirHarkonnen · 2026-02-20T20:19:34Z

Goal

Connection reuse improves efficiency for both the client and server side. Coupled with support for SSL session reuse and advanced HTTP features, this implementation reduces the computing cost of implementing encryption in HTTP trackers.
Hopefully, this will stimulate more trackers to adopt encryption.

Breaking changes

None; disabled by default. Compile time opt-in.

Breaking changes (with respect to the reference http_connection_tracker implementation):

send_host_in_connect is ignored because it is required for curl's HTTP/1.x proxy and cannot be turned off. I suggest removing the setting and always turning it on.
proxy_hostnames is ignored for HTTP proxies, because it is required in curl's HTTP proxy implementation.

Relevant curl issues:

Keep-alive pings for connections in the connection cache are not processed, causing idle connections to die (https://github.com/curl/curl/blob/master/docs/TODO.md#monitor-connections-in-the-connection-pool).
No port-hint support for HTTPS-rr DNS records feature advertisement (ECH, ALPN) (HTTPS RR ip hint and port ignored curl/curl#17045):
- HTTPS-rr is currently experimental in libcurl and can only be enabled/disabled at compile time. It should be disabled until a baseline of features (such as port hint) is supported. For example, trackers can use a port hint to upgrade from HTTP to TLS connections on a different port.
- Only works with libcurl DoH and c-ares resolvers (c-ares does not support ECH a.t.m.)

Questions:

It is unclear how to run all automatic test with two HTTP stacks. Should all tests be executed for both?
libtorrent/aux_/deadline_timer.hpp uses system_clock, which is not resistant to time jumps. Any chance of changing it to steady_clock?

Unit test coverage:

Building with curl=on allows running all current tests for the curl HTTP stack.
New test are added for ssrf and ip_filter because the simulator tests are not run yet (maybe remove these in favor of running the simulator tests?).
New test are added for intrusive_list, bitmask and curl.
No test are added for curl specific features like HTTP/2. Proper functioning of curl internals is assumed.
Simulator support is not added yet
Can be implemented by stubbing the curl_pool, and redirecting the curl_easy_* calls to secretly use the http_connection class. Is this desired? should this be a separate PR to keep this one small?

Testing:

A rebased version has been tested with a custom qbittorrent docker image, based on libtorrent v2.0.11.
https://github.com/TheBaronVladimirHarkonnen/docker-qbittorrent-nox
- Compiled with curl_debug=on all the HTTP connection information is logged. Observed that the connections are being reused.
- When the application is shutting down, the "stopped" events are sent. Good, the application exits cleanly.
- Tested that curl errors are propagated properly to the UI.
All testcases are passing with curl=on, no degradation in test running times.
Did not run the simulator.
Has not been tested on Windows yet, I will start working on setting up a windows box.

Implementation design:

Uses libcurl thread-unaware multi library with boost asio async io handlers (single threaded, on the main libtorrent thread)
All the newly added algorithms are O(1) per HTTP request, using O(N) to process N requests. However, the libcurl multi interface uses naive algorithms with O(N^2) time complexity (inside libcurl for queuing and list operations).

arvidn · 2026-02-21T10:14:01Z

please simplify this patch. also, review it to make sure you don't duplicate any functionality. also, make sure you have CI test coverage of the new functionality.

arvidn · 2026-02-21T10:15:22Z

btw. I would think the main issue with tracker announces right now is that there's no support for pipelining. If multiple torrents use the same tracker, they shouldn't each have to connect and SSL handshake.

TheBaronVladimirHarkonnen · 2026-02-21T15:35:18Z

btw. I would think the main issue with tracker announces right now is that there's no support for pipelining. If multiple torrents use the same tracker, they shouldn't each have to connect and SSL handshake.

You make a good point, I agree that this is both easy to implement in boost and cleaner to integrate into the current code base. The reason why I'm more interested in curl is as follows:

With HTTP/1.1, it is possible to have more announces than you can process using a single connection. For example, a tracker expects an announce every 30 minutes, when an announce takes 0.5 seconds, you can only support at most 30*60*2=3600 torrents. H2 or H3 can solve this with doing many parallel announces, and the server can choose how many streams it wants to use.
With HTTP/3 and QUIC, a server is no longer limited by the number of available TCP ports (0-65536), allowing it to keep more persistent connections.
Encrypted Hello support, while this is currently not working yet. This is a cool privacy upgrade.
HTTPS-RR support, making clients manually upgrade their tracker urls to HTTPS is not feasible. HTTP redirect for upgrades are not feasible. I want a server to upgrade all my torrents to HTTPS by simply adding a HTTPS-RR DNS record.
boost beast has no development roadmap and has been in maintenance mode for years. I doubt it will ever support more advanced HTTP features. Writing correct/good code is their main goal, not low level efficiency.

I'm not happy with the current state of curl but it is the best library out there. Thinking about the future is important too, will the simple boost implementation be good enough 10 years from now? I would like the think we are all using the new advanced HTTP features by then.

Don't be afraid to reject the integration of curl if you don't like the idea, I would totally understand.

TheBaronVladimirHarkonnen · 2026-02-27T23:56:18Z

just a status update, I'm still busy reworking this.

…k request filter into shared common code, simplifications

TheBaronVladimirHarkonnen · 2026-03-15T21:57:05Z

needs more work, don't review yet.

seabashed · 2026-05-31T17:26:59Z

needs more work, don't review yet.

Looking forward to whenever you have time for this. The fact we need to create and tear down a connection with all the overhead (especially SSL announces) that entails is quite sad. Hope to see modern approaches adopted in our favorite tools! 👍

TheBaronVladimirHarkonnen · 2026-06-02T08:35:03Z

@seabashed
Thank you for the motivation to work on this.

Currently I'm working on creating a custom HTTP request queuing system to avoid using the curl queuing system entirely. The curl queuing was not made for applications like Libtorrent and we are better of not using it.

This queuing creates a separate queue per tracker:

HTTP server without connection reuse, then use up to f(queue_size) connections, e.g. 5 connections per 200 queued items would avoid overloading the server.
HTTP1.X server with connection reuse, then use up to g(queue_size) connections, e.g. 1 connection per 200 queued items.
HTTP2/HTTP3 server with connection reuse, then use up to k(queue_size) connections, e.g. 1 connection with many streams would work.

The balance is between not creating too many connections, but at the same time being able to send all announces at startup/shutdown in a reasonable time-frame.

seabashed · 2026-08-29T10:07:32Z

Thanks @TheBaronVladimirHarkonnen, is this still worth pursuing or effectively replaced by @arvidn's work on 9654c51?

(opt-in) use curl for tracker communication

39f5cee

TheBaronVladimirHarkonnen added 4 commits February 24, 2026 12:13

curl http stack: refactor common code

9e1f9c6

curl http stack: split up request class

fa95765

curl http stack: simplify by removing cache variable

5c8ac14

curl http stack: simplify by removing async_wait optimization

89ec7e9

TheBaronVladimirHarkonnen added 4 commits March 9, 2026 18:26

curl http stack: implement redirects, implement custom timeout, rewor…

e26fd2f

…k request filter into shared common code, simplifications

curl http stack: refactor redirect/timeout to request abstraction level.

5b81e60

curl http stack: match project include guard style

fe0bba4

curl http stack: refactoring

b0da104

TheBaronVladimirHarkonnen marked this pull request as draft March 15, 2026 21:19

Navigation Menu

Uh oh!

(opt-in) use curl for tracker communication - #8159

(opt-in) use curl for tracker communication#8159
TheBaronVladimirHarkonnen wants to merge 9 commits into
arvidn:masterfrom
TheBaronVladimirHarkonnen:master_curl

TheBaronVladimirHarkonnen commented Feb 20, 2026 •

edited

Loading

Uh oh!

arvidn commented Feb 21, 2026

Uh oh!

arvidn commented Feb 21, 2026

Uh oh!

TheBaronVladimirHarkonnen commented Feb 21, 2026 •

edited

Loading

Uh oh!

TheBaronVladimirHarkonnen commented Feb 27, 2026

Uh oh!

TheBaronVladimirHarkonnen commented Mar 15, 2026

Uh oh!

seabashed commented May 31, 2026

Uh oh!

TheBaronVladimirHarkonnen commented Jun 2, 2026

Uh oh!

seabashed commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

TheBaronVladimirHarkonnen commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Goal

Breaking changes

Relevant curl issues:

Questions:

Unit test coverage:

Testing:

Implementation design:

Uh oh!

arvidn commented Feb 21, 2026

Uh oh!

arvidn commented Feb 21, 2026

Uh oh!

TheBaronVladimirHarkonnen commented Feb 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TheBaronVladimirHarkonnen commented Feb 27, 2026

Uh oh!

TheBaronVladimirHarkonnen commented Mar 15, 2026

Uh oh!

seabashed commented May 31, 2026

Uh oh!

TheBaronVladimirHarkonnen commented Jun 2, 2026

Uh oh!

seabashed commented Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TheBaronVladimirHarkonnen commented Feb 20, 2026 •

edited

Loading

TheBaronVladimirHarkonnen commented Feb 21, 2026 •

edited

Loading